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Evaluating the Design, Implementation, and Impact of a Non-Traditional Cohort 
Ed.D. Program in Educational Leadership and Policy Analysis 

R.V. Hatley, D.E. Arredondo, J.F. Donaldson, P.M. Short & L.W. Updike 

In recent years university programs that prepare individuals for professional practice in law, 
medicine, business administration, and school leadership have been heavily criticized. Questions have 
been raised about whether universities can continue as the most appropriate environment for 
professional preparation (Forsyth, 1993). Leaders within these fields of study share a common 
commitment to developing practitioners who demonstrate thoughtful performances in complex 
situations of practice. They also share the common challenge of designing, implementing, and 
evaluating programs of study that integrate sophisticated levels of knowledge of content and deep 
understanding of the strategies and skills required for solving intricate problems in practice. 

The practice-content balance debate continues among and between professors and practitioners 
in these fields. For example, Pitner (1988) noted that like education, medical, business, and legal 
training have all been criticized for their lack of relevance and usefulness. Schon (1987) 
characterized training programs of most professions as suffering from problems of abstract and 
irrelevant training, problems compounded by a failure to test theories from the field to guide practice. 

In response to the plethora of criticism, professional practice preparation programs are being 
changed. Program reform has been limited, however, by employing competence models within settings 
poorly suited to fundamental change. The'XJentral flaw of competence models is their underlying 
assumption that professional performance is an individual affair (Nowlen, 1988). Professionals do 
not work alone, but rather in rich social contexts that involve relationships with, for example, peers, 
the organization, members of other professions, and professional associations. As a consequence, 
despite a decade of national debate challenging the professions to restructure their preparatory 
programs, much change has been gradual, incremental, and minimal. The same is true for education. 

In fact, Forsyth (1993, p. 329) writes, “Although there is some frantic program revision going on in 
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educational administration preparation, much of it is not genuine experimentation. . . Much of it is 
very bad stuff, ill conceived adaptations of reform ideas turned perverse.” Bjork and Ginsberg 
(1995) also argue that fundamental structural change in leadership preparation programs within the 
traditional university setting is unlikely to occur, that only “in developing settings” will the true 
restructuring called for in the reform debate be nurtured. 

In September 1995, the Department of Educational Leadership and Policy Analysis, University 
of Missouri-Columbia, began an innovative executive Ed.D. program, collaboratively designed around a 
performance model and delivered in a “developing setting.” The program is (1) grounded on the notion 
that professional practice can be improved through a continuous reflective learning cycle, linking 
practice with the program theory and connecting new knowledge with reflection in and on practice; (2) 
designed to exhibit principles of cognitive learning theory applied within the contexts of social and 
cultural experiences of adult learners; (3) structured within a student cohort framework to promote 
the development of supportive, collegial learning communities; and (4) designed to take full advantage 
of the students’ work sites as rich laboratories for the study of professional practice. The curriculum 
is structured within four thematic focus blocks, organized as seamless sets of inquiry rather than 
traditional courses. Instructional strategies include collaborative engagement, problem-based 
learning, and interdisciplinary approaches to problem solving, all with authentic assessments. 
Integrated into each thematic unit are activities that develop students’ knowledge and skills in research 
design, data acquisition and analysis, and program evaluation (Hatley, Arredondo, Donaldson & Phillips, 
1996). Unlike most educational administrator preparation programs, this program includes 
individuals from K-12 public schools, postsecondary education institutions, governmental agencies, 
and corporations with education-related missions. This multiple-level focus enriches the experiences 
of a diverse student population, draws upon faculty expertise across disciplines, and reflects the 
interconnectedness among educational sectors (Arredondo, Donaldson, Hatley & Updike, 1996). 
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Project Purposes and Methods 



Although many professional preparation programs have been reformed and numerous studies of 
those reforms conducted, few have included comprehensive on-going evaluations of the curricular 
components. Fewer still have focused on the broad impact of those components, including effects on 
practice. This project begins such an examination of the program at the University of Missouri by 
focusing on selected perceptions of the effects of instructional Themes 1 and 2 (Fall Semester 1995 and 
Winter Semester 1996). In this phase of the program evaluation, we will examine cohort member and 
educational leadership faculty and administrator perceptions about: (1) how the development of new 
knowledge structures and skills begin to influence the learner’s practices within the work setting; (2) 
how a learner’s ability to cognitively process knowledge in reflective, self-regulated ways, changes 
over the course of her or his participation in a three-year doctoral program; (3) how educational 
leaders acquire the essential knowledge and skills if they are to effectively coach and mentor 
collaborative work teams, analyze and solve vexing problems in complex organizations, and develop, 
implement, and evaluate educational programs and change processes; (4) how practitioner scholarship 
is affected through an integrated approach to learning about and practicing educational research, and 
(5) if and how a university’s “traditional on-campus” doctoral program is affected by faculty 
involvement with a very different and innovative off-campus program. 

Data Collection an d Analysis . Described below are the data collection and analysis procedures 
used in this preliminary examination of selected perceptions of Theme 1 and Theme 2 effects. As 
previously noted, this represents only the first two phases of a multi-year action research project that 
will, over the course of three years, assess the procedural components, outcomes, and impact of the 
program across four content themes, two research skill development components, two applied research 
practicums, and the students’ one-year dissertation projects. Multiple data collection methods will be 
used in each phase to provide sequential, discrete, and aggregate assessments and to allow for eventual 
triangulation of data for the purposes of summative evaluation of the doctoral program outcomes. 



Data reported here were collected from three sources: (1) cohort program students, (2) 
cohort faculty, and (3) other faculty and administrators from the regular on-campus program in 
educational leadership. Each data source is described separately. Cohort students were asked to 
complete the Cohort Survey Instrument, calling for 10-point scaled rating responses across four 
assessment dimensions in each of eight knowledge and skill content areas covered in Theme 1 , Creating 
Personal Mastery and Effective Learning Teams, and ten knowledge and skill content areas covered in 
Theme 2, Creating Effective Learning Organizations. One dimension measured the level of knowledge 
and skill of the cohort participant prior to Theme 1 and Theme 2 instruction; another, the level of 
knowledge and skills after completion of the semester-long instruction, collaborative problem solving, 
and independent reflection activities. From these two sets of data, “growth” measures were calculated 
by comparing pre-participation levels of knowledge and skills to post-participation levels. To obtain 
data necessary to examine the impact of cohort participation on students’ acquisition of knowledge and 
leadership skills, the students responded to survey items which measured the extent to which they 
believe their participation in instructional theme activities and events contributed to their growth and 
development. Finally, students were asked to indicate the level to which they have applied Theme 1 and 
Theme 2 knowledge and skills in their educational practices within their unique work settings. 

Data were analyzed using descriptive statistics, such as scale score ranges, mathematical 
averages, and standard deviations for the cohort’s responses. Explanatory details about the wording of 
the survey instrument items, response scaling, and analysis of the data are provided as footnotes to 
Table 1 (for Theme 1) and to Table 2 (for Theme 2). Cohort students were also given the opportunity 
to provide candid, expansive responses to open-ended items which were included as a part of the Cohort 
Survey Instrument. Examples of stimulus items are as follow: “I would describe being a member of 

the cohort doctoral program as . The best part of being in the cohort has been . If I could 

have changed one thing about the cohort in Theme 1 [Theme 2] 2, it would have been . The most 

beneficial thing I learned during Theme 1 [Theme 2] was . The most irrelevant thing I learned 
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was . What I liked best about the instruction in Theme 1 [Theme 2] was . What I liked 

least about the instruction was . ” Students’ responses were subjected to content analysis to 

determine central themes and intensity of expressed views and are discussed later from naturalistic 
inquiry and program evaluation perspectives. Data analyses, results, and interpretative discussions 
are provided separately for the two 1995-96 academic year cohort program instructional themes. 

An interview guide was developed to collect data from the University of Missouri-Columbia 
Educational Leadership and Policy Analysis faculty and campus administrators in regard to how much 
they know about the off-campus cohort doctoral program, their perceptions of the relative successes of 
the program to date, the perceived impact of the program on campus program changes, and on their own 
instructional strategies. Fourteen faculty members and administrators were interviewed and their 
scripted responses were subjected to content analysis to determine central themes and intensity of 
expressed views. In the discussion section of this report the data are described both as aggregated data 
and as categorical data based on the extent to which individual faculty members have been “actively 
engaged” in the cohort program, ranging from extensive engagement to little or no engagement. Faculty 
engagement possibilities included participation on the program design team, participation in the cohort 
screening and selection processes, contributing member of the instructional delivery team, contact and 
discussions with members of the cohort students, and/or, at the lowest engagement level, “what I know 
I have heard about at department faculty meetings and some informal discussions with key players.” 

Results: Theme 1, Creating Personal Mastery and Effective Learning Teams 

Student Data: Theme 1 Survey Instrument. Scaled-Response Items 

Growth for the cohort during Theme 1 , semester one, is evident in all eight knowledge and skill 
concept areas, as measured by the Cohort Survey Instrument. (See Table 1.) The lowest pre-theme 
levels of knowledge and skill were in the areas of reflective practice and cognitive learning and adult 
learning theory, the highest, team building and verbal and nonverbal communication. Interestingly, 



the two lowest pre-theme areas of knowledge and skill were rated as the highest areas of growth by the 
cohort students after Theme 1 instruction, collaborative engagement events, and reflective practice 
activities were completed. The two highest pre-theme areas of knowledge and skill were rated as the 
lowest growth areas by the students at the end of the theme. 

Increments of aggregate cohort group growth in the concept areas, attributed by the students to 
their participation in the first semester of the cohort doctoral program ranged from lows of 70.50% 
for team building and 70.90% for research understanding and use to highs of 85.26% for cognitive 
learning and adult learning theory and 89.51% for reflective practice. The data, presented in Table 1, 
also indicate clearly that the cohort students found ways to apply the knowledge obtained and the skills 
developed during Theme 1 in their professional work settings. Means for application of knowledge and 
skills ranged from a low of 6.27 for research understanding and use (indicating a moderate level of 
application) to a high of 8.65 for verbal and nonverbal communication (indicating a high level of 
application). Therefore, it appears that the students are finding opportunities to address their own and 
their unique educational organizations’ needs for developing personal mastery and creating effective 
learning teams. 

Student Data: Theme 1 Survey Instrument. Open-Ended Response Items 

Student responses to the open-ended survey items were first compiled by item and then grouped 
around the emergent themes. The data groups were then collapsed to eliminate redundancies. This 
resulted in four major categories of responses: (1) student feelings about the cohort itself and about 
being a member of the cohort, (2) student perceptions of their own and others’ responses to the 
content knowledge presented in Theme 1: Creating Personal Mastery and Effective Team Learning, (3) 
student responses to the instructional methods used in Theme 1 , and (4) over-all reflections about the 
cohort program as experienced in Theme 1. Each category is discussed separately following Table 1. 
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Table 1 



Knowledge Base and Leadership Skills. Cohort Growth and Application: 
Theme 1 - Creating Personal Mastery and Effective Learning Teams * 



A. 

Knowledae & Skill 


B. 

Pre-Theme 


C. D. 

Post-Theme Growth in 


E. F. 

% Growth Application 

Attributed by Student 

to Program, in the Work 
if + Growth Setting 
(1-10 Scale) (1-10 Scale) 


Concept Areas 


Knowledge Knowledge Knowledge 

and Skills, and Skills, and Skills, 

by Concept by Concept by Concept 

(1-10 Scale) (1-10 Scale) (B - C = D) 


Covered in Cohort 
Theme 1 (1995 
Fall Semester) 


1. Coanitive Structuring 










and Development 












Low Rating 


2.00 


5.00 


-1.00 


60.00 


3.00 


High Rating 


8.00 


9.00 


5.00 


100.00 


10.00 


Cohort Average 


4.83 


7.39 


2.57 


81.78 


7.70 


Standard Deviation 1.70 


1.16 


1.47 


10.30 


1 .49 


2. Coanitive Learnina and 










Adult Learnina Theorv 










Low Rating 


2.00 


6.00 


0.00 


70.00 


5.00 


High Rating 


8.00 


9.00 


5.00 


100.00 


10.00 


Cohort Average 


4.65 


7.65 


3.00 


85.26 


7.61 


Standard Deviation 1.56 


0.93 


1.28 


0.79 


1 .37 


3. Verbal and Nonverbal 










Communications 












Low Rating 


2.00 


7.00 


0.00 


10.00 


7.00 


High Rating 


10.00 


10.00 


6.00 


100.00 


10.00 


Cohort Average 


7.09 


8.70 


1.61 


76.14 


8.65 


Standard Deviation 1 .83 


0.82 


1.59 


24.10 


1.03 


4. Interpersonal and Intra- 










oraanizational Conflict 










Low Rating 


2.00 


7.00 


0.00 


10.00 


5.00 


High Rating 


10.00 


10.00 


6.00 


100.00 


10.00 


Cohort Average 


5.96 


8.04 


2.09 


78.30 


7.91 


Standard Deviation 1 .87 


1.02 


1.70 


19.20 


1.20 


5. Coachina and Mentorina 










in Learnina Oraanizations 










Low Rating 


3.00 


7.00 


0.00 


40.00 


6.00 


High Rating 


10.00 


10.00 


5.00 


100.00 


10.00 


Cohort Average 


6.64 


8.55 


1.91 


80.52 


8.32 


Standard Deviation 1.76 


0.91 


1.23 


16.20 


1 .21 



Continued 
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Table 1 (continued) 



A. 

Knowledae & Skill 


B. 

Pre-Theme 


C. 

Post-Theme 


D. 

Growth in 


E. 

% Growth 


F. 

Application 

h\/ Student 


Concept Areas 


Knowledge Knowledge Knowledge 

and Skills, and Skills, and Skills, 

by Concept by Concept by Concept 

(1-10 Scale) (1-10 Scale) (B - C = D) 


Attributed 


Covered in Cohort 
Theme 1 (1995 
Fall Semester) 


to Program, in the Work 
if + Growth Setting 
(1-10 Scale) (1-10 Scale) 


6. Team Building 

in Learninq Oraanizations 

Low Rating 4.00 


7.00 


0.00 


30.00 


3.00 


High Rating 


10.00 


10.00 


3.00 


100.00 


10.00 


Cohort Average 


7.23 


8.64 


1.41 


70.50 


8.55 


Standard Deviation 


1.45 


0.90 


1.05 


20.60 


1.50 


7. Reflective Practice 


Low Rating 


1.00 


5.00 


0.00 


60.00 


4.00 


High Rating 


10.00 


10.00 


9.00 


100.00 


10.00 


Cohort Average 


4.09 


7.91 


3.82 


89.51 


8.00 


Standard Deviation 


2,37 


1.41 


2.15 


10.90 


1.35 


8. Research Under- 
standina and Use 
Low Rating 


1 .00 


5.00 


0.00 


10.00 


1.00 


High Rating 


8.00 


9.00 


6.00 


100.00 


10.00 


Cohort Average 


5.09 


7.14 


2.05 


70.90 


6.27 


Standard Deviation 


2.02 


1.08 


1.50 


26.50 


2.27 



Reliability analysis for the instrument (40 scaled items): Alpha coefficient = .7901 



'Notes: Information about collecting and analyzing the data in Table 1. 

Column A: Knowledge and Skills Concept Areas 

Eight areas of declarative knowledge, procedural knowledge, and skills included in Theme 1 

Column B: Pre-Theme Knowledge and Skills, instrument item 
Item - My level of knowledge and related skills as of July 1, 1995: 

Data - Ten-point scaled responses, 1 = None 10 = Extensive 

Column C: Post-Theme Knowledge and Skills, instrument item 

Item - My level of knowledge and related skills as of January 24, 1996: 

Data - Ten-point scaled responses, 1 = None 10 = Extensive 
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Notes to Table 1 (continued) 



Column D: Growth in Knowledge and Skill during Theme 1 

Data - Computed by subtracting respondents’ Post-Theme scores from Pre-Theme scores 
Formula - (Column B - Column C = Growth) - computed for each individual respondent 

Column E: Percent of Growth Attributable to Cohort Program Participation 

Item - The degree to which I believe Theme 1 is responsible for my level of knowledge 

and related skills as of January 24, 1996: 

Data - Ten-point scaled responses, 1 = 0-10% 10 = 100% 

Column F: Applying Theme 1 Knowledge and Skills in the Work Place 

Item - My current level of application of knowledge and related skills in my work setting: 

Data - Ten-point scaled responses, 1 = None 10 = Extensive 



Student feelings about the cohort and about membership in the cohort . Students described their 
feelings about being a member of the cohort in overwhelmingly positive terms. They saw the 
experience as both “challenging and exhilarating” and as “an excellent learning experience.” One said 
that the cohort provided “an unusual opportunity to break with tradition and to approach learning with 
real world application in mind” while another wrote that s/he found the cohort “extremely important 
to my professional growth”. One cohort member described him/herself as “one of many seeking a 
common goal; [the program is] an experiment that is stressful, exciting, hard work, a struggle at 
times, not so bad at other times, and [overall] a privilege.” 

Students saw collaboration as the best part of the cohort experience. They wrote about the 
opportunity to learn from peers, sharing experiences and ideas, building relationships, networking, 
the synergy of the group experience, and the support provided from others. One described Theme 1 as 
providing a “truly collaborative setting. Instructors are part of the ‘community of learners’ and the 
cohort members bring varied and rich professional experience. The result is that we all benefit far 
more than any of us would in a traditional instructional approach.” 
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Everything about membership in the cohort was not viewed as positive, however. Student’ 
responses about what they would have changed about the cohort produced comments about a need to 
establish group norms early on, to work on building trust, to provide structures that allow for more 
one-on-one time with professors, to “muzzle” the constant complainers, to be allowed to work with 
only one group, and to have the primary work group be composed of voluntary members rather than 
assigned. One student expressed these ideas as, “We needed to do more to develop cohort norms and a 
high level of trust early in the semester. Because many of the cohort members were confident, 
comfortable, and able to trust almost immediately, we allowed the group to overlook the significant 
needs of a handful of members who seemed unable to get over this very significant hurdle. Their 
limitations became limitations for the entire group.” Another, arguing for a different structure for 
small groups, wrote, “allow groups to form (naturally) and remain intact. In doing whole group 
discussion, we could all learn from the large group, but for building relationships (of support and 
need) it would be best if we could be in the same group continuously. In any organization you work with 
the large organizational group but you depend on the same small group within a department for work on 
a daily basis.” 

Student reactions to the content of Theme 1 . The content knowledge selected for major emphasis 
during Theme 1 was that required for creating personal mastery and effective team learning. This 
language was borrowed from Peter Senge (1990). By it is meant the understandings and skills required 
for educational leaders to become self-regulated continuous learners and reflective practitioners, and 
the knowledge and skills needed to facilitate the continuous growth and development of the adults (both 
as individuals and teams) with whom they work so that those adults continuously function as self- 
regulated learners and reflective practitioners. Student responses indicated that their knowledge about 
reflective practice as well as their abilities to function as reflective practitioners has grown 
considerably during the Theme 1 learning experiences. Students described these understandings and 
skills as the most beneficial ideas learned during Theme 1 . Sample comments illustrating this point 
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included a note about the importance of reflective practice and collaboration skills, “I now regularly 
apply both concepts, the first for myself and the second for my organization.” A second cohort member 
stated that, “Before I start questioning others about their actions, I’m learning to look at myself.” 

Still another said the most beneficial idea was “team building -- (especially)how to work effectively 
with people whom I do not necessarily see eye to eye with.” 

As could be predicted from their ratings of knowledge gained, students described considerable 
growth in their own knowledge in several sub-units within this theme, particularly those related to 
reflective practice. For example, one student said that his/her current capacity for personal mastery 
was “greatly improved -- the greatest growth in my professional development has come in the area of 
personal mastery. I’m better equipped at looking at myself objectively and [at asking] how my actions 
are affecting the organization.” Another said these skills were “evolving - I learned much about 
myself, particularly in the reflective journal, but still have much to learn.” Others described their 
growth with such terms as “developing”, “greatly improved”, “significantly increased”, “stretched”, 
“quality improved, quantity increased” and “forever setting new boundaries”. 

Students were somewhat less sure about growth in their capacity to create effective learning 
teams. They said, “improved, but still needing work” and “somewhat increased; my confidence in this 
area has grown through my interaction with my team members. I will be more assertive as future 
teams develop.” Others expressed caution about their own growth in this capacity. For example, one 
cohort member wrote that s/he felt, “More capable, but [I] still have reservations about skills 
required to be truly effective.” 

Students were also asked to identify the most irrelevant ideas learned during Theme 1 . Most had 
difficulty with this item and responded that “nothing was irrelevant”, that “everything was 
applicable” and one said, somewhat humorously, that the procedure for managing dinner fit into this 
category. Two identified the group project as less relevant, however; and one described the 
collaborative action research project as irrelevant. 
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Student reactions to the instructional methods used in Theme 1 . Cohort members reported that 



they liked the instructional methods used by the professors, particularly the workshop approach 
(Arredondo and Rucinski, 1994). The workshop approach, as used in Theme 1, involved both 
structured and unstructured reflective journaling, patterned responses to journal entries, individual 
and group conferences with the professors, structured small group discussions of specific tasks, 
projects structured by individual work groups with periodic progress reports to peers, and production 
of a final product that demonstrated each task group’s understanding of a significant portion of the 
theme content knowledge. Specific aspects of the workshop approach were also described as 
praiseworthy because they led to meaningful learning. For example, collaborative and interactive 
learning activities, group work, and reflective journaling were mentioned by several students as 
leading to “real learning”. One student wrote that, the “professors eventually led us to a very 
meaningful group experience and project.” Another cohort member identified “the opportunity to 
interact within a diverse group” as what s/he like best about the instruction in Theme 1. This student 
added, “I learned quickly that some of my preconceived notions were totally in error. My group was 
diverse and my experience in that group was enriching. I think I’m a better educator for it.” 

While most cohort member comments about instructional methods were extremely positive, 
some students reported that they liked least “the lectures that we did have,” “the older and 
[sometimes] outdated journal articles,” and “the lack of structure with the group project.” One 
student wrote that the “lack of written material to provide an overall framework” caused confusion at 
times, another said that “instructors dispensed information as if we didn’t have any prior knowledge” 
and still another would have preferred more student input into the instructional design. 

Students’ overall reflections about the cohort program as experienced in Theme 1 . Most of the 
reflections offered by the cohort members were extremely positive. Three typical comments serve to 
illustrate this interpretation. 
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Theme 1 was an exceptional experience. I am excited to be in a program that brings immediate 
relevance to learning. I am convinced that the cohort delivery model is a vast improvement over 
traditional graduate delivery systems. It is wonderful to see graduate professors with the courage 
to step out from under the shadows of their ivory towers and really be about the business of 
developing real world educational leaders. It is also important to recognize that high academic 
expectations and a customer friendly graduate school delivery model are not mutually exclusive 
concepts. I would not be in graduate school if my only option was a traditional on-campus program. 
My obligations to my organization take precedence over my obligation to the university. The cohort 
program is the first graduate program that I have experienced that recognizes this reality and I 
appreciate it. The result is that both I and my organization benefit. 

My participation in the structured learning from Theme 1 reinforced my intuitive beliefs about 
how people learn — but more than that, the theme gave me the language to discuss collaborative, 
structured, and participative learning and why it works. My learning has given me the background 
to be a meaningful participant in the discussions we are having at the post secondary level regarding 
how we teach. I also have used my learning in structuring group activities that I have facilitated -- 
I believe the power of the group to come with its own answers. 

I enjoyed Theme 1 because of the workshop approach. I felt as though my thoughts and feelings 
had some validity and were heard. I seem to learn better when given time to reflect through my 
feelings and understandings rather than through a more structured journal approach. I feel I have 
been able to work well with the team members of my project group because it fit better with my 
style of learning. For once the curriculum and assignments were adjusted to my needs rather than 
me having to adjust my learning style to the needs of the curriculum and the professor. 

Negative student reflections about the overall cohort as experienced in Theme 1 were rare, and 
when offered, were mild by comparison. For example, one student wrote, “With all the concerns 
mentioned above, please keep in mind that this is better than it might have been. However, with all 
the build-up prior to the class beginning, it is less than my expectations.” Another said, “Overall, the 
experience in Theme 1 was positive. I learned a lot and grew in knowledge greatly -- especially in the 
area of interpersonal skills. As I reflect on Theme 1 , I am left with the perception that some cohort 
members acted very selfishly, never truly bonded with anyone, and did not respect the facilitators.” 



Results: Theme 2, Creating Effective Learning Organizations 

Student Data: Theme 2 Survey Instrument. Scaled-Response Items 

Growth for the cohort during Theme 2, semester two, is evident in all ten knowledge and skill 
concept areas, as measured by the Cohort Survey Instrument. (See Table 2 and discussion following.) 
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Table 2 



Knowledge Base and Leadership Skills. Cohort Growth and Application: 
Theme 2 - Creating Effective Learning Organizations * 



A. 

Knowledae & Skill 


B. 

Pre-Theme 


C. 

Post-Theme 


D. 

Growth in 


E. 

% Growth 


F. 

Application 


ConceDt Areas 


Knowledge Knowledge Knowledge 

and Skills, and Skills, and Skills, 

by Concept by Concept by Concept 

(1-10 Scale) (1-10 Scale) (B - C = D) 


Attributed by Student 

to Program, in the Work 
if + Growth Setting 
(1-10 Scale) (1-10 Scale) 


Covered in Cohort 
Theme 2 (1996 
Winter Semester) 


1 . Analysis of Oraani- 












zational Situations 












Low Rating 


2.00 


7.00 


1.00 


70.00 


7.00 


High Rating 


8.00 


9.00 


7.00 


100.00 


10.00 


Cohort Average 


4.83 


8.39 


3.57 


87.80 


8.30 


Standard Deviation 


1.85 


0.72 


1.78 


9.50 


0.93 


2. Structural Nature 












of Oraanizations 












Low Rating 


1.00 


7.00 


0.00 


50.00 


6.00 


High Rating 


9.00 


10.00 


7.00 


100.00 


10.00 


Cohort Average 


5.26 


8.57 


3.30 


85.70 


8.36 


Standard Deviation 


2.12 


0.84 


2.08 


13.40 


1.14 


3. Human Resources and 










Group Empowerment 










Low Rating 


3.00 


7.00 


0.00 


30.00 


7.00 


High Rating 


10.00 


10.00 


6.00 


100.00 


10.00 


Cohort Average 


6.26 


8.91 


2.65 


79.10 


8.91 


Standard Deviation 


1.96 


0.79 


1 .72 


20.90 


0.85 


4. Open Systems Nature 










of Oraanizations 












Low Rating 


2.00 


6.00 


0.00 


10.00 


3.00 


High Rating 


8.00 


10.00 


7.00 


100.00 


10.00 


Cohort Average 


4.30 


7.91 


3.61 


80.90 


7.78 


Standard Deviation 


2.08 


1 .04 


2.06 


22.30 


1 .51 


5. Political Nature 












of Oraanizations 












Low Rating 


2.00 


7.00 


0.00 


50.00 


4.00 


High Rating 


8.00 


10.00 


6.00 


100.00 


10.00 


Cohort Average 


5.35 


8.52 


3.17 


83.50 


8.35 


Standard Deviation 


2.08 


1.04 


1.85 


15.00 


1 .40 



Continued 
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Table 2 (continued) 



A. 


B. 


C. 


D. 


E. 


F. 


Knowledae & Skill Pre-Theme 


Post-Theme 


Growth in 


% Growth 


Application 


Concept Areas Knowledae 


Knowledge 


Knowledge 


Attributed 


by Student 


Covered in Cohort and Skills, 


and Skills, 


and Skills, 


to Program, 


in the Work 


Theme 1 (1995 by Concept 


by Concept 


by Concept 


if + Growth 


Setting 


Fall Semester) (1-10 Scale) (1-10 Scale) 


(B - C = D) 


(1-10 Scale) (1-10 Scale) 


6. Svmbolic Nature 












of Organizations 












Low Rating 


2.00 


7.00 


1.00 


40.00 


6.00 


High Rating 


8.00 


10.00 


7.00 


100.00 


10.00 


Cohort Average 


4.55 


8.36 


3.82 


86.40 


8.23 


Standard Deviation 


1.90 


0.90 


1.71 


13.30 


1 .23 


7. Uniaue Characteristics of 










Learning Organizations 










Low Rating 


2.00 


7.00 


0.00 


10.00 


6.00 


High Rating 


9.00 


10.00 


7.00 


100.00 


10.00 


Cohort Average 


5.00 


8.50 


3.59 


81.80 


8.32 


Standard Deviation 


1.98 


0.96 


2.09 


20.40 


1 .09 


8. Team Buildina and Collab- 










orative Problem Solving 










Low Rating 


3.00 


7.00 


0.00 


10.00 


7.00 


High Rating 


10.00 


10.00 


6.00 


100.00 


10.00 


Cohort Average 


7.09 


8.86 


1.77 


69.00 


8.86 


Standard Deviation 


1.90 


0.83 


1.51 


26.20 


0.89 


9. Reflective Practice 


in and 










about Organizational Settings 










Low Rating 


2.00 


6.00 


0.00 


10.00 


5.00 


High Rating 


10.00 


10.00 


6.00 


100.00 


10.00 


Cohort Average 


5.64 


8.18 


2.55 


75.50 


7.77 


Standard Deviation 


2.26 


1 .01 


1.90 


24.40 


1 .34 


lO.Usina. Conducting, and 










Reporting Research 










Low Rating 


2.00 


4.00 


0.00 


10.00 


2.00 


High Rating 


8.00 


10.00 


8.00 


100.00 


10.00 


Cohort Average 


4.77 


7.68 


2.91 


75.50 


6.73 


Standard Deviation 


1.82 


1.46 


2.1 1 


24.40 


2.25 



Reliability Analysis for the Instrument (50 scaled items): Alpha Coefficient = .8837 
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•Notes: Information about collecting and analyzing the data in Table 2. 



Column A: Knowledge and Skills Concept Areas 

Ten areas of declarative knowledge, procedural knowledge, and skills included in Theme 2 

Column B: Pre-Theme Knowledge and Skills, instrument item 

Item - My level of knowledge and related skills as of January 1, 1996: 

Data - Ten-point scaled responses, 1 = None 10 = Extensive 

Column C: Post-Theme Knowledge and Skills, instrument item 
Item - My level of knowledge and related skills as of May 31, 1996: 

Data - Ten-point scaled responses, 1 = None 10 = Extensive 

Column D: Growth in Knowledge and Skill during Theme 2 

Data - Computed by subtracting respondents’ Post-Theme scores from Pre-Theme scores 
Formula - (Column B - Column C = Growth) - computed for each individual respondent 

Column E: Percent of Growth Attributable to Cohort Program Participation 

Item - The degree to which I believe Theme 2 is responsible for my level of knowledge 

and related skills as of May 31, 1996: 

Data - Ten-point scaled responses, 1=0-10% 10 = 100% 

Column F: Applying Theme 2 Knowledge and Skills in the Work Place 

Item - My current level of application of knowledge and related skills in my work setting: 

Data - Ten-point scaled responses, 1 = None 10 = Extensive 



The lowest pre-theme levels of knowledge and skill were in the areas of Analysis of 
Organizational Situations, Symbolic Nature of Organizations, and Open Systems Nature of 
Organizations: the highest areas were Team Building and Collaborative Problem Solving and Human 
Resources Nature of Organizations. It is not surprising that Team Building and Collaborative Problem 
Solving was viewed by the students as their Theme 2 pre-instruction highest ability area, given that 
the focus of Theme 1 had been on improving personal mastery and effective team building. As expected, 
and hoped, the standard deviations for post-theme knowledge and skill levels were all lower than those 
for the pre-theme measures, indicating that the group variance in knowledge and skill declined as a 
function of the learning experiences. 

Increments of aggregate cohort group growth in the ten concept areas, attributed by the students 
to their participation in the second semester of the cohort doctoral program, were distributed in the 
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following growth attribution ranges: (1) from 60% to 69.9% for knowledge and skills in Team 
Building and Collaborative Problem Solving: (2) from 70% to 79.9% for the study areas of Human 
Resources and Group Empowerment, Reflective Practice in and about Organizational Settings, and 
Using, Conducting, and Reporting Research; and (3) above 80% for knowledge and skills in Analysis of 
Organizational Organizations, Open Systems Nature of Organizations, Political Nature of Organizations, 
Structural Nature of Organizations, Symbolic Nature of Organizations, and Unique Characteristics of 
Learning Organizations. 

Data presented in Table 2 also indicate the extent to which students have found ways to apply in 
their professional work settings the knowledge obtained and the skills developed during Theme 2. 
Means for application of knowledge and skills (on a 10-point scale, where 1= no application and 
10=extensive application) ranged from a low of 6.73 for Using, Conducting, and Reporting Research to 
a high of 8.91 for Human Resources and Group Empowerment. Students indicated a high level of 
knowledge and skill application (above 8.0 on the scale) in the work setting for seven of Theme 2’s ten 
instructional domains. Cohort students clearly are finding opportunities to apply their new knowledge 
and skills in assisting their own education work sites’ attempts to become more effective learning 
organizations. However, we do not know whether these opportunities are arising because of the 
individual student’s increased confidence in her/his new knowledge and skills, an already established 
(pre-cohort program) reputation for assertive and effective leadership, and/or superordinates’ 
recognition of the student’s increased contribution potential because of participation in the educational 
leadership cohort doctoral program. These possible explanations will be explored during interviews 
with work-site peers and superordinates during subsequent phases of this program evaluation project. 

Student Data: Theme 2 Survey Instrument. Open-Ended Response Items 

Student responses to the open-end survey items in the Theme 2 survey were compiled by 
item and then grouped around the emergent themes. After collapsing data groups to eliminate 
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redundancies, four major categories of responses, consistent with the four categories identified 
in the first survey, emerged: (1) student feelings about the cohort itself and about being a 
member of the cohort, (2) student perceptions of their own and others responses to the content 
knowledge presented in Theme 2, Creating Effective Learning Organizations, (3) student 
responses to the instructional methods used in Theme 2, and (4) over-all reflections about the 
cohort program. Each Theme 2 student reaction category is discussed separately below. 

Student feelings about the cohort and about membership in the cohort . Students 
continued to describe their feelings about being a member of the cohort in very positive terms. 
They described their participation as “invigorating,” “exhilarating,” and “very worthwhile” 
and noted that their “confidence is building.” Collaboration in Theme 2 was a significant part of 
the cohort experience for many students. One commented about the “enjoyment of learning with 
my peers and working together in finding solutions.” Another saw personal benefit in 
“learning more about how my own learning/working style fits with the style of others, both in 
my organization and in the cohort.” One student reflected the observations of several cohort 
members by stating that the best part of the cohort continued to be “spending time with talented 
and committed people on common goals.” This perception about the benefits of collaboration 
included instructors as well as students. Appreciation for the “camaraderie (among) the staff 
and the students of the cohort” and “the opportunity to discuss and seek feedback from 
instructors” was also a prevalent theme found in results of the Theme 2 surveys. 

Some negative observations about cohort membership were also evident. One cohort 
member requested “more consistent feedback on my progress.” Another felt that the final 
report was flawed, noting that while “guidelines are needed, they cannot be too confining.” 

One student was concerned that “we still seem to establish guidelines after assignments are 
turned in” and another concluded, “we all need to move from our comfort zones and work with 
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all members/groups. Our strength will come from working with others - no isolationism!” 

Student concerns surfaced about two concurrent group projects required during Theme 
2. Students indicated that they did not do as well as they might have on either project because 
they did not have sufficient time to successfully focus on either one. One student asked that 
instructors assign only “one project team at a time! Another indicated that “there was not 
enough time to do both well.” Instructors were asked to “revamp project teams” and “require 
less project work” or provide “more time in class to do group work.” One student believed the 
cohort needed to keep “doing meaningful projects. We just need more specific expectations.” 

Student responses to the content of Theme 2 . The content knowledge selected for major 
emphasis during Theme 2 was that required for creating effective learning organizations. This 
theme was approached using perspectives provided by Lee Bolman and Terrence Deal ( 1991) 
and Peter Senge ( 1990). Student responses indicated that their knowledge about learning 
organizations has grown during their Theme 2 experiences. Students specifically noted the 
value of looking at organizations using Bolman and Deal’s metaphor of “frames” to describe 
different perspectives. One stated that the most beneficial idea learned was “ analysis of 
problems using the frames. This is something I will use extensively as I examine problems and 
challenges in my own workplace.” Other students commented that the best thing about Theme 2 
was learning about “organizational frames,” “organizations and their leadership”, and “the 
multiple prospective approach to addressing difficult issues.” 

Students described extensive growth in their personal knowledge during Theme 2. 
Illustrative of the comments of many students was the observation “I learned a lot during this 
theme. The things I learned will help me tremendously on my job.” A student stated “This 
theme moved my thinking about the analysis of organizations forward significantly.” Another 
reported that “The work required was much more rigorous this semester. Once I grasped how to 
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approach each frame I felt comfortable with the material. I learned a great deal.” 

Students were asked to identify the most irrelevant thing learned during Theme 2. 
Several students responded by stating “nothing in particular,” “no learning is irrelevant,” 
and “nothing was really irrelevant.” Several students commented that time spent dealing with 
human resources, hiring, and interviewing was irrelevant, and two others commented about the 
irrelevance of the information provided by several outside speakers. One student brought up a 
recurring issue by commenting humorously about the “continued challenge of establishing a 
cheap dinner.” 

Student responses to the instructional methods used in Theme 2 . Comments made by 
cohort members demonstrated an overall positive response to the instructional methods used 
during this theme. Many students were able to identify specific aspects of instruction that they 
found to be beneficial to their individual learning. One appreciated “the multiple instructional 
perspectives used during this theme” and another appreciated “knowing up front what was 
expected.” More specifically, others identified “fishbowl exercises”, “large group discussions 
involving peers and instructional staff,” “small group learning experiences,” and “short 
personal feedback conferences” as activities they liked best about instruction in this theme. 

Also noted as beneficial were the “group project and individual problem based learning” 
and “powerful reading assignments linked to good case studies.” Several students mentioned 
the value of an outstanding textbook and the approach of studying the content through the use of 
individual “frames.” One cohort member concluded that the instructional approach used in 
Theme 2 “allowed me to focus on my own growth rather than ‘jumping’ through someone else’s 
contrived hoops.” 

There were some aspects of Theme 2 instruction that were seen by cohort members as 
less than positive. Several students expressed concern about the lack of reflective journaling 
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that had been a prevalent part of the Theme I experience. One student thought that Theme 2 had 
become “too traditional” because there was” no emphasis on reflection” and another noted that 
there was “no structured opportunity for personal or group reflection.” Other students 
disliked lecture. One stated “I don’t dislike lecture altogether, I just like it least,” and another 
student felt there was “too much lecture before group work.” One student raised a concern about 
“some rather unfocused, unchallenging discussions,” but also noted that “This is a challenge, 
getting discussion to be penetrating and really focused on deep issues.” 

One widespread concerned surfaced. Many students found the inclusion of a series of 
guest speakers in Theme 2 to be of limited benefit. One commented that “some of the outside 
‘experts’ did not link as well as they might have to analytical frames we were studying.” 

Another stated bluntly “eliminate the guest speakers.” and a third characterized them as 
“interesting, but not much of an addition to my overall learning.” One student made essentially 
the same comment and then concluded, “I think that our time might have been better spent 
using the expertise within the cohort rather than asking external experts to speak to us.” 

Students’ overall reflections about the cohort program as experienced in Theme 2 . The 
vast majority of comments made by cohort members about Theme 2 were positive. One student 
concluded” I learned a lot during this theme. The things I learned will help me tremendously on 
my job.” Students also indicate that they felt a “strong sense of ‘group’ and less apprehensive 
than in the first theme,” and that they “were beginning to assimilate team concepts.” One 
student described Theme 2 as: “Ambiguous, always changing, interesting, and challenging. Not 
knowing what to expect is hard. It’s the double edged sword of a new program. I like the 
innovation and adaptability, yet wish I knew what was coming up next. Better organization with 
the assignments has alleviated this issue somewhat this semester.” Finally, another student 
described the cohort as “exhilarating on the good days and frustrating on the bad days. The 
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length of the program will cause people to be up and down about the group, the work load, etc. I 
still fine the program very exciting.” 



Results: Faculty Interviews Regarding Cohort Program Impact on Campus 

Data from the faculty interviews were grouped based on the level of involvement with the cohort 
as highly involved, moderately involved, and uninvolved. Highly involved faculty are those who 
professed to know a great deal about the cohort. They helped plan the cohort program and were part of 
the instructional team for Theme 1 . Moderately involved faculty are those who generally professed to 
have some limited knowledge about the cohort program. Typically they were involved in some planning 
for the cohort and assisted in the screening of candidates for admission to the program. Uninvolved 
faculty indicated that they had little knowledge of the program and had not participated in any way; 
regular faculty meeting reports were their primary sources of information. 

Highly involved faculty . Four faculty members and administrators were identified as having 
been highly involved in the cohort program. They participated actively in planning and developing the 
program, in the screening and selection process, and three of them delivered the majority of the 
instruction for Theme 1. The fourth had minimal instructional involvement but had other 
opportunities for direct face-to-face contact with all cohort members. 

Interview responses from the highly involved faculty were overwhelmingly positive. They 
were obviously knowledgeable about the cohort endeavor and believed that it was having direct impact 
on their own instructional practices on campus. Each of them was able to cite changes in specific 
instructional strategies as a result of their participation in the team teaching effort. These faculty 
found team teaching to be a refreshing change from what one of them described as the traditional 
isolation experienced by faculty on campus. This faculty member characterized his own personal 
change as “monumental”. Another described a growing tendency toward collaboration among what had 
been the former “higher education” and “educational administration” faculty members. While more 
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cautious in judging the impact on campus programs, the highly involved faculty did believe that the 
cohort program had enriched a developing dialogue among the entire faculty about what constitutes the 
best leadership practices and how best to deliver leadership preparation programs. 

Moderately Involved Faculty . Four faculty members and administrators were identified as 
moderately involved in the cohort program. Their involvement varied, however. They all had at least 
some opportunity to have direct contact with all the cohort members, with two assisting in the 
screening and selection of the students and two participating in both the screening and selection process 
as well as having a brief involvement with the instruction for Theme 1 . 

In terms of effects on professional practices, the responses of the moderately involved faculty 
and administrators also varied. Two indicated that the program has had no impact on their own 
practices; one reported that her brief experience with the program had expanded her ability to 
contribute to current on-campus planning efforts. A fourth stated that the cohort program was moving 
toward “consistency with my prior practices,” and saw the cohort program as having a positive 
influence on current on-campus change efforts. 

The statements from the moderately involved faculty and administrators about the impact of the 
cohort program on campus also showed differences of opinion. Two stated that little or no impact has 
been felt on campus while the other two clearly noted that the cohort program is influencing current 
discussions about how to change the on-campus doctoral program. 

Uninvolved Faculty . Seven faculty members and administrators were identified as uninvolved in 
the cohort program. The knowledge they had about the program had been gained from reports at faculty 
meetings and from informal conversations with members of the other two groups. Three faculty 
participated in the screening and selection process while two of these three had also met briefly with 
members of the cohort prior to the beginning of Theme 1 . The remaining four of the seven members in 
this group had been completely uninvolved in the cohort program. Two of the seven members of this 
group stated unequivocally that the cohort had had no impact on their professional practices. The other 
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five all either stated directly or implied uncertainty about the ways in which the program may have 
influenced their practices. These five made a number of statements that indicated that even though they 
were unsure regarding specifics about the program, they were highly interested in knowing more about 
what was occurring. Some of their comments demonstrated a desire to know more about how cohort 
efforts might benefit their own instructional practices. One said of the program, “It’s like learning a 
new language,” and another noted that, “This makes me think more about ‘walking my talk’.” 

Interviewed faculty and administrators raised a number of questions and concerns. Primary 
among these were issues concerning resources. For example, how will the department deal with the 
impact on already scarce resources? Because the cohort program takes faculty members off campus on 
a regular basis, many of those interviewed felt that it will be difficult for the department to continue to 
successfully meet on-campus program requirements. Additionally a number of faculty members 
expressed concerns about the sheer drain of energy on the professors who make a commitment to travel 
to Kansas City on a weekly basis -- while still maintaining their on-campus responsibilities. One 
wondered how such commitments may affect performance on the more traditionally rewarded activities. 
A number Of faculty members also raised concerns about how the department would handle thirty 
dissertations at one time. 

A few faculty members raised questions or issues about the diversity of the cohort, especially in 
terms of prior preparation and experience of the students selected for the program. One wondered, 
“Have we cast the net too wide?” Some asked whether such a diverse group could work successfully 
together within a cohort experience. Several asked about the opinions and attitudes of the cohort 
members, seemingly concerned about how well students have accepted the cohort delivery model. 

Synthesis: Discussion, Implications, and Conclusions 

This impact evaluation project was designed to provide preliminary information about the 
effects of a new educational leadership preparation program currently being delivered by the 
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University of Missouri-Columbia to a cohort of Kansas City area students pursuing an Ed.D. degree. 
Perceptual data about the program effects were collected from cohort program participants, cohort 
faculty, and from other Educational Leadership and Policy Analysis faculty and campus administrators. 

Both quantitative and qualitative data were collected from cohort students using the Cohort 
Survey Instrument. The students used a 10-point scale to rate the four assessment dimensions in each 
of the eight knowledge and skill content areas presented in Theme 1 and in each of the ten knowledge and 
skill content areas presented in Theme 2. The four assessment dimensions included before and after 
Theme 1 and Theme 2 assessments of students’ knowledge and skills in each content area, the degree to 
which participation in the program contributed to their growth and development, and the level to which 
they applied knowledge and skills from the themes in their educational practices within their work 
settings. The cohort students also responded to ten open-ended items about their cohort membership, 
the content knowledge and skills presented in themes one and two, the instructional methods, and 
overall reflections about their learning experiences. 

The evaluation efforts, to date, have been limited to perceptual, self-reported data from cohort 
students and therefore lack checks of validity from other data sources and through other data collection 
procedures. Nonetheless, the project’s results for evaluating themes one and two of this innovative, 
off-campus cohort doctoral program provide evidence for the following preliminary conclusions: 

• The program has had positive impact on students’ cognitive growth as well as on their 

professional practice. 

• Content, instructional strategies, and experiential learning activities for both Theme 1 and Theme 2 
were well received by students. Students feel free to express both their positive and negative 
comments about the program, with the belief that they can influence both the content and context of 
their own learning within the doctoral program. 
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• Faculty members’ and campus administrators’ perceptions of the program’s impact on their own 
practice and on the department’s on-campus instructional programs vary with their level of 
involvement with the cohort program. 

Theme 1 (1995 fall semester) of the program focused on Creating Personal Mastery and 
Effective Learning Teams, Theme 2 (1996 winter semester) on Creating Effective Learning 
Organizations. Theme 3 for the 1996 fall semester focuses on Creating the Content, Context, and 
Climate for Effective Learning and Theme 4 for the 1 997 winter semester will deal with Creating 
Practice and Policy for Addressing Critical Issues. Summer sessions for 1 996 and 1 997 engage the 
cohort students in knowledge and skill development related to measurement, statistics, research design, 
and program evaluation. Authentic assessment methods will be used for the students’ doctoral 
comprehensive exams during the 1 997 summer session. The 1 997-98 academic year has been 
designated as the “dissertation year” with a target date for graduation projected for August 1998. 

Future plans for continued program evaluation call for a multifaceted approach to data collection 
to achieve the evaluation purposes outlined earlier in the paper. Because the program is performance- 
based, the approach to any summative program evaluation will require the merging of perceptual data 
with actual performance-based data. These data will be collected from cohort students, from their 
supervisors and professional colleagues, and/or through actual observation of the students’ on-the-job 
performance. Future data collection will also focus on the impact the cohort program has had on 
university faculty members, on campus administrators, and on the department’s instructional delivery 
strategies and graduate degree programming for preparing educational leaders. 

The form and content of future evaluation efforts are still in the design stage. We assert that, 
just as the cohort program has responded to calls for reform in professional preparation programs, 
so too must the evaluation approach be responsive to and consistent with a performance-based design. 
This will require innovation in the program evaluation approaches employed, potentially leading to 
new understandings and models of how reformed professional programs can and should be evaluated. 
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